{
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "# 7 文档转换器\n",
    "加载文档后，您通常需要转换它们以更好地适合您的应用程序。\n",
    "\n",
    "最简单的示例是，您可能希望将长文档拆分为可以放入模型上下文窗口的较小块。\n",
    "\n",
    "LangChain 有许多内置的文档转换器，可以轻松拆分、组合、过滤和以其他方式操作文档。\n",
    "\n"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "当您想要处理长文本时，有必要将该文本拆分为块。\n",
    "\n",
    "听起来很简单，但这里有很多潜在的复杂性。\n",
    "\n",
    "理想情况下，您希望将语义相关的文本片段放在一起。\n",
    "\n",
    "“语义相关”的含义可能取决于文本的类型。本笔记本展示了实现此目的的几种方法。"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "默认推荐的文本拆分器是 RecursiveCharacterTextSplitter。此文本拆分器采用字符列表。它尝试基于第一个字符的拆分来创建块，但如果任何块太大，它就会移动到下一个字符，依此类推。默认情况下，它尝试拆分的字符是 [\"\\n\\n\", \"\\n\", \" \", \"\"]"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "page_content='Madam Speaker, Madam Vice President, our First Lady and Second Gentleman. Members of Congress and of' metadata={'start_index': 0}\n",
      "page_content='of Congress and of Congress and the Cabinet. Justices of the Supreme Court. My fellow Americans.' metadata={'start_index': 82}\n"
     ]
    }
   ],
   "source": [
    "# This is a long document we can split up.\n",
    "with open('documentstore/state_of_the_union.txt') as f:\n",
    "    state_of_the_union = f.read()\n",
    "\n",
    "from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
    "text_splitter = RecursiveCharacterTextSplitter(\n",
    "    # Set a really small chunk size, just to show.\n",
    "    chunk_size = 100,\n",
    "    chunk_overlap  = 20,\n",
    "    length_function = len,\n",
    "    add_start_index = True,\n",
    ")\n",
    "texts = text_splitter.create_documents([state_of_the_union])\n",
    "print(texts[0])\n",
    "print(texts[1])"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-08-26T08:22:53.058951200Z",
     "start_time": "2023-08-26T08:22:50.918628500Z"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "page_content='Madam Speaker, Madam Vice President, our First Lady and Second Gentleman. Members of Congress and of' metadata={'document': 1, 'start_index': 0}\n"
     ]
    }
   ],
   "source": [
    "#下面是将元数据与文档一起传递的示例，请注意，元数据与文档一起拆分。\n",
    "metadatas = [{\"document\": 1}, {\"document\": 2}]\n",
    "documents = text_splitter.create_documents([state_of_the_union, state_of_the_union], metadatas=metadatas)\n",
    "print(documents[0])"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-08-26T08:23:13.169127400Z",
     "start_time": "2023-08-26T08:23:13.152289Z"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "outputs": [
    {
     "data": {
      "text/plain": "'Madam Speaker, Madam Vice President, our First Lady and Second Gentleman. Members of Congress and of'"
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text_splitter.split_text(state_of_the_union)[0]"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-08-26T08:23:56.133999200Z",
     "start_time": "2023-08-26T08:23:56.079442700Z"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "outputs": [
    {
     "data": {
      "text/plain": "['cpp',\n 'go',\n 'java',\n 'js',\n 'php',\n 'proto',\n 'python',\n 'rst',\n 'ruby',\n 'rust',\n 'scala',\n 'swift',\n 'markdown',\n 'latex',\n 'html',\n 'sol']"
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# CodeTextSplitter 允许您使用多种语言支持拆分代码。导入枚举 Language 并指定语言。\n",
    "from langchain.text_splitter import (\n",
    "    RecursiveCharacterTextSplitter,\n",
    "    Language,\n",
    ")\n",
    "# Full list of support languages\n",
    "[e.value for e in Language]"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-08-26T08:24:25.828056600Z",
     "start_time": "2023-08-26T08:24:25.808051600Z"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "outputs": [
    {
     "data": {
      "text/plain": "['\\nclass ', '\\ndef ', '\\n\\tdef ', '\\n\\n', '\\n', ' ', '']"
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# You can also see the separators used for a given language\n",
    "RecursiveCharacterTextSplitter.get_separators_for_language(Language.PYTHON)"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-08-26T08:24:28.595909Z",
     "start_time": "2023-08-26T08:24:28.555383500Z"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "outputs": [
    {
     "data": {
      "text/plain": "[Document(page_content='def hello_world():\\n    print(\"Hello, World!\")', metadata={}),\n Document(page_content='# Call the function\\nhello_world()', metadata={})]"
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#下面是一个使用 PythonTextSplitter 的示例\n",
    "PYTHON_CODE = \"\"\"\n",
    "def hello_world():\n",
    "    print(\"Hello, World!\")\n",
    "\n",
    "# Call the function\n",
    "hello_world()\n",
    "\"\"\"\n",
    "python_splitter = RecursiveCharacterTextSplitter.from_language(\n",
    "    language=Language.PYTHON, chunk_size=50, chunk_overlap=0\n",
    ")\n",
    "python_docs = python_splitter.create_documents([PYTHON_CODE])\n",
    "python_docs"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-08-26T08:25:01.694951Z",
     "start_time": "2023-08-26T08:25:01.675944300Z"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "outputs": [
    {
     "data": {
      "text/plain": "[Document(page_content='function helloWorld() {\\n  console.log(\"Hello, World!\");\\n}', metadata={}),\n Document(page_content='// Call the function\\nhelloWorld();', metadata={})]"
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#下面是一个使用 JS 文本拆分器的示例\n",
    "JS_CODE = \"\"\"\n",
    "function helloWorld() {\n",
    "  console.log(\"Hello, World!\");\n",
    "}\n",
    "\n",
    "// Call the function\n",
    "helloWorld();\n",
    "\"\"\"\n",
    "\n",
    "js_splitter = RecursiveCharacterTextSplitter.from_language(\n",
    "    language=Language.JS, chunk_size=60, chunk_overlap=0\n",
    ")\n",
    "js_docs = js_splitter.create_documents([JS_CODE])\n",
    "js_docs"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-08-26T08:25:25.888226200Z",
     "start_time": "2023-08-26T08:25:25.833221Z"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "outputs": [
    {
     "data": {
      "text/plain": "[Document(page_content='# 🦜️🔗 LangChain', metadata={}),\n Document(page_content='⚡ Building applications with LLMs through composability ⚡', metadata={}),\n Document(page_content='## Quick Install\\n\\n```bash', metadata={}),\n Document(page_content=\"# Hopefully this code block isn't split\", metadata={}),\n Document(page_content='pip install langchain', metadata={}),\n Document(page_content='```', metadata={}),\n Document(page_content='As an open source project in a rapidly developing field, we', metadata={}),\n Document(page_content='are extremely open to contributions.', metadata={})]"
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#下面是使用 Markdown 文本拆分器的示例。\n",
    "markdown_text = \"\"\"\n",
    "# 🦜️🔗 LangChain\n",
    "\n",
    "⚡ Building applications with LLMs through composability ⚡\n",
    "\n",
    "## Quick Install\n",
    "\n",
    "```bash\n",
    "# Hopefully this code block isn't split\n",
    "pip install langchain\n",
    "```\n",
    "\n",
    "As an open source project in a rapidly developing field, we are extremely open to contributions.\n",
    "\"\"\"\n",
    "md_splitter = RecursiveCharacterTextSplitter.from_language(\n",
    "    language=Language.MARKDOWN, chunk_size=60, chunk_overlap=0\n",
    ")\n",
    "md_docs = md_splitter.create_documents([markdown_text])\n",
    "md_docs"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-08-26T08:25:44.339578800Z",
     "start_time": "2023-08-26T08:25:44.315579900Z"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "page_content='Madam Speaker, Madam Vice President, our First Lady and Second Gentleman. Members of Congress and of' metadata={}\n",
      "page_content='of Congress and of Congress and the Cabinet. Justices of the Supreme Court. My fellow Americans.' metadata={}\n"
     ]
    }
   ],
   "source": [
    "# 按字符递归拆分\n",
    "from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
    "# This is a long document we can split up.\n",
    "with open('documentstore/state_of_the_union.txt') as f:\n",
    "    state_of_the_union = f.read()\n",
    "text_splitter = RecursiveCharacterTextSplitter(\n",
    "    # Set a really small chunk size, just to show.\n",
    "    chunk_size = 100,\n",
    "    chunk_overlap  = 20,\n",
    "    length_function = len,\n",
    ")\n",
    "texts = text_splitter.create_documents([state_of_the_union])\n",
    "print(texts[0])\n",
    "print(texts[1])"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-08-26T09:09:17.741802Z",
     "start_time": "2023-08-26T09:09:17.730804900Z"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "outputs": [
    {
     "data": {
      "text/plain": "['Madam Speaker, Madam Vice President, our First Lady and Second Gentleman. Members of Congress and of',\n 'of Congress and of Congress and the Cabinet. Justices of the Supreme Court. My fellow Americans.']"
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text_splitter.split_text(state_of_the_union)[:2]"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-08-26T09:09:23.997378Z",
     "start_time": "2023-08-26T09:09:23.976372800Z"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Madam Speaker, Madam Vice President, our First Lady and Second Gentleman. Members of Congress and of Congress and the Cabinet. Justices of the Supreme Court. My fellow Americans.\n"
     ]
    }
   ],
   "source": [
    "#按令牌拆分\n",
    "from langchain.text_splitter import CharacterTextSplitter\n",
    "with open('documentstore/state_of_the_union.txt') as f:\n",
    "    state_of_the_union = f.read()\n",
    "text_splitter = CharacterTextSplitter.from_tiktoken_encoder(\n",
    "    chunk_size=100, chunk_overlap=0\n",
    ")\n",
    "texts = text_splitter.split_text(state_of_the_union)\n",
    "print(texts[0])"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-08-26T09:10:56.207434200Z",
     "start_time": "2023-08-26T09:10:51.651228700Z"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Madam Speaker, Madam Vice President, our\n"
     ]
    }
   ],
   "source": [
    "#我们也可以直接加载一个tiktoken拆分器\n",
    "from langchain.text_splitter import TokenTextSplitter\n",
    "\n",
    "text_splitter = TokenTextSplitter(chunk_size=10, chunk_overlap=0)\n",
    "\n",
    "texts = text_splitter.split_text(state_of_the_union)\n",
    "print(texts[0])"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-08-26T09:11:12.135896800Z",
     "start_time": "2023-08-26T09:11:12.082528300Z"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "无论您的模型的体系结构如何，当您包含 10+ 检索到的文档时，性能都会大幅下降。\n",
    "\n",
    "简而言之：当模型必须在长上下文中访问相关信息时，往往会忽略提供的文档。"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "outputs": [
    {
     "data": {
      "text/plain": "Downloading (…)e9125/.gitattributes:   0%|          | 0.00/1.18k [00:00<?, ?B/s]",
      "application/vnd.jupyter.widget-view+json": {
       "version_major": 2,
       "version_minor": 0,
       "model_id": "a9541a60af8443769ecfc1d6681492e7"
      }
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": "Downloading (…)_Pooling/config.json:   0%|          | 0.00/190 [00:00<?, ?B/s]",
      "application/vnd.jupyter.widget-view+json": {
       "version_major": 2,
       "version_minor": 0,
       "model_id": "172dd576f8354090974d200e25fe797e"
      }
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": "Downloading (…)7e55de9125/README.md:   0%|          | 0.00/10.6k [00:00<?, ?B/s]",
      "application/vnd.jupyter.widget-view+json": {
       "version_major": 2,
       "version_minor": 0,
       "model_id": "76c6809e598f475197306bcdb20b07f4"
      }
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": "Downloading (…)55de9125/config.json:   0%|          | 0.00/612 [00:00<?, ?B/s]",
      "application/vnd.jupyter.widget-view+json": {
       "version_major": 2,
       "version_minor": 0,
       "model_id": "8c90932a854f45268d7694ec7e8692f6"
      }
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": "Downloading (…)ce_transformers.json:   0%|          | 0.00/116 [00:00<?, ?B/s]",
      "application/vnd.jupyter.widget-view+json": {
       "version_major": 2,
       "version_minor": 0,
       "model_id": "b1f157c7d7774a30ac8526d3f814b0b1"
      }
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": "Downloading (…)125/data_config.json:   0%|          | 0.00/39.3k [00:00<?, ?B/s]",
      "application/vnd.jupyter.widget-view+json": {
       "version_major": 2,
       "version_minor": 0,
       "model_id": "4d036863f19148f5b49b6bb888aed8db"
      }
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": "Downloading pytorch_model.bin:   0%|          | 0.00/90.9M [00:00<?, ?B/s]",
      "application/vnd.jupyter.widget-view+json": {
       "version_major": 2,
       "version_minor": 0,
       "model_id": "7c8e1d0e971b408abc90c5b19ef88d61"
      }
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": "Downloading (…)nce_bert_config.json:   0%|          | 0.00/53.0 [00:00<?, ?B/s]",
      "application/vnd.jupyter.widget-view+json": {
       "version_major": 2,
       "version_minor": 0,
       "model_id": "f108a268e8074c94a5244a242336a738"
      }
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": "Downloading (…)cial_tokens_map.json:   0%|          | 0.00/112 [00:00<?, ?B/s]",
      "application/vnd.jupyter.widget-view+json": {
       "version_major": 2,
       "version_minor": 0,
       "model_id": "cd32a91f34a1447bae353291bec7ae5c"
      }
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": "Downloading (…)e9125/tokenizer.json:   0%|          | 0.00/466k [00:00<?, ?B/s]",
      "application/vnd.jupyter.widget-view+json": {
       "version_major": 2,
       "version_minor": 0,
       "model_id": "cf5f2929e09a420b9f0f9fabf29b5c3a"
      }
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": "Downloading (…)okenizer_config.json:   0%|          | 0.00/350 [00:00<?, ?B/s]",
      "application/vnd.jupyter.widget-view+json": {
       "version_major": 2,
       "version_minor": 0,
       "model_id": "1ef113c558884504876a468f1d0a3cf9"
      }
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": "Downloading (…)9125/train_script.py:   0%|          | 0.00/13.2k [00:00<?, ?B/s]",
      "application/vnd.jupyter.widget-view+json": {
       "version_major": 2,
       "version_minor": 0,
       "model_id": "026376196a0b4cd1807a2e93b7acc295"
      }
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": "Downloading (…)7e55de9125/vocab.txt:   0%|          | 0.00/232k [00:00<?, ?B/s]",
      "application/vnd.jupyter.widget-view+json": {
       "version_major": 2,
       "version_minor": 0,
       "model_id": "2bddbe83425347a0a9c4036d908da3a8"
      }
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": "Downloading (…)5de9125/modules.json:   0%|          | 0.00/349 [00:00<?, ?B/s]",
      "application/vnd.jupyter.widget-view+json": {
       "version_major": 2,
       "version_minor": 0,
       "model_id": "dca40da422344303a526f00a0eea6eb9"
      }
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": "[Document(page_content='This is a document about the Boston Celtics', metadata={}),\n Document(page_content='The Celtics are my favourite team.', metadata={}),\n Document(page_content='L. Kornet is one of the best Celtics players.', metadata={}),\n Document(page_content='The Boston Celtics won the game by 20 points', metadata={}),\n Document(page_content='Larry Bird was an iconic NBA player.', metadata={}),\n Document(page_content='Elden Ring is one of the best games in the last 15 years.', metadata={}),\n Document(page_content='Basquetball is a great sport.', metadata={}),\n Document(page_content='I simply love going to the movies', metadata={}),\n Document(page_content='Fly me to the moon is one of my favourite songs.', metadata={}),\n Document(page_content='This is just a random text.', metadata={})]"
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 导入必要的库和模块\n",
    "import os\n",
    "import chromadb\n",
    "from langchain.vectorstores import Chroma\n",
    "from langchain.embeddings import HuggingFaceEmbeddings\n",
    "from langchain.document_transformers import LongContextReorder\n",
    "from langchain.chains import StuffDocumentsChain, LLMChain\n",
    "from langchain.prompts import PromptTemplate\n",
    "from langchain.llms import OpenAI\n",
    "\n",
    "# 获取预训练的词嵌入\n",
    "embeddings = HuggingFaceEmbeddings(model_name=\"all-MiniLM-L6-v2\")\n",
    "\n",
    "# 定义一组文本文档\n",
    "texts = [\n",
    "    \"Basquetball is a great sport.\",\n",
    "    \"Fly me to the moon is one of my favourite songs.\",\n",
    "    \"The Celtics are my favourite team.\",\n",
    "    \"This is a document about the Boston Celtics\",\n",
    "    \"I simply love going to the movies\",\n",
    "    \"The Boston Celtics won the game by 20 points\",\n",
    "    \"This is just a random text.\",\n",
    "    \"Elden Ring is one of the best games in the last 15 years.\",\n",
    "    \"L. Kornet is one of the best Celtics players.\",\n",
    "    \"Larry Bird was an iconic NBA player.\",\n",
    "]\n",
    "\n",
    "# 使用这些文本文档和预训练的词嵌入创建一个检索器\n",
    "retriever = Chroma.from_texts(texts, embedding=embeddings).as_retriever(\n",
    "    search_kwargs={\"k\": 10}\n",
    ")\n",
    "\n",
    "# 定义一个查询\n",
    "query = \"What can you tell me about the Celtics?\"\n",
    "\n",
    "# 基于这个查询检索相关的文档，并按相关性分数排序\n",
    "docs = retriever.get_relevant_documents(query)\n",
    "docs\n"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-08-26T09:18:55.359152500Z",
     "start_time": "2023-08-26T09:18:15.709829Z"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "outputs": [
    {
     "data": {
      "text/plain": "[Document(page_content='The Celtics are my favourite team.', metadata={}),\n Document(page_content='The Boston Celtics won the game by 20 points', metadata={}),\n Document(page_content='Elden Ring is one of the best games in the last 15 years.', metadata={}),\n Document(page_content='I simply love going to the movies', metadata={}),\n Document(page_content='This is just a random text.', metadata={}),\n Document(page_content='Fly me to the moon is one of my favourite songs.', metadata={}),\n Document(page_content='Basquetball is a great sport.', metadata={}),\n Document(page_content='Larry Bird was an iconic NBA player.', metadata={}),\n Document(page_content='L. Kornet is one of the best Celtics players.', metadata={}),\n Document(page_content='This is a document about the Boston Celtics', metadata={})]"
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 使用LongContextReorder对文档进行重新排序\n",
    "# 较不相关的文档将被放在列表的中间，而较相关的文档将被放在开始和结束的位置\n",
    "reordering = LongContextReorder()\n",
    "reordered_docs = reordering.transform_documents(docs)\n",
    "\n",
    "# 确认最相关的4篇文档位于开始和结束的位置\n",
    "reordered_docs\n"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-08-26T09:20:29.258547300Z",
     "start_time": "2023-08-26T09:20:29.229995900Z"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "outputs": [
    {
     "data": {
      "text/plain": "'\\n\\nThe Celtics are a professional basketball team based in Boston, Massachusetts. They have won 17 NBA Championships, the most out of any team in the league. They have had many iconic players over the years, such as Larry Bird, and currently have stars such as Kemba Walker and Jayson Tatum. They are currently one of the top teams in the league and are looking to make a deep playoff run this season.'"
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 重写提示\n",
    "# 创建一个文档提示，只是简单地返回文档内容\n",
    "document_prompt = PromptTemplate(\n",
    "    input_variables=[\"page_content\"], template=\"{page_content}\"\n",
    ")\n",
    "document_variable_name = \"context\"\n",
    "\n",
    "# 初始化OpenAI模型\n",
    "llm = OpenAI()\n",
    "\n",
    "# 定义一个自定义的提示，它会提示模型基于给定的文本（上下文）回答以下的问题\n",
    "stuff_prompt_override = \"\"\"给定以下文本：\n",
    "-----\n",
    "{context}\n",
    "-----\n",
    "请回答以下问题：\n",
    "{query}\"\"\"\n",
    "\n",
    "# 使用上述自定义提示创建一个提示模板\n",
    "prompt = PromptTemplate(\n",
    "    template=stuff_prompt_override, input_variables=[\"context\", \"query\"]\n",
    ")\n",
    "\n",
    "# 实例化LLMChain，这是一个简单的链，只使用OpenAI模型和上述的提示模板\n",
    "llm_chain = LLMChain(llm=llm, prompt=prompt)\n",
    "\n",
    "# 实例化StuffDocumentsChain，它使用llm_chain来处理查询，并将文档作为上下文提供\n",
    "chain = StuffDocumentsChain(\n",
    "    llm_chain=llm_chain,\n",
    "    document_prompt=document_prompt,\n",
    "    document_variable_name=document_variable_name,\n",
    ")\n",
    "\n",
    "# 运行StuffDocumentsChain，使用重新排序的文档作为上下文，并提供查询\n",
    "chain.run(input_documents=reordered_docs, query=query)\n"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-08-26T09:21:57.975611500Z",
     "start_time": "2023-08-26T09:21:54.862988700Z"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## 文本嵌入模型\n",
    "嵌入类是设计用于与文本嵌入模型接口的类。有很多嵌入模型提供程序（OpenAI，Cohere，Hugging Face等） - 这个类旨在为所有它们提供一个标准接口。\n",
    "\n",
    "嵌入创建一段文本的矢量表示形式。这很有用，因为这意味着我们可以考虑向量空间中的文本，并执行诸如语义搜索之类的操作，在其中我们寻找向量空间中最相似的文本片段。\n",
    "\n",
    "LangChain 中的基本嵌入类公开了两种方法：一种用于嵌入文档，另一种用于嵌入查询。前者将多个文本作为输入，而后者则采用单个文本。将它们作为两个独立方法的原因是，某些嵌入提供程序对文档（要搜索）和查询（搜索查询本身）具有不同的嵌入方法。"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "outputs": [
    {
     "data": {
      "text/plain": "(5, 1536)"
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from langchain.embeddings import OpenAIEmbeddings\n",
    "\n",
    "embeddings_model = OpenAIEmbeddings()\n",
    "\n",
    "embeddings = embeddings_model.embed_documents(\n",
    "    [\n",
    "        \"Hi there!\",\n",
    "        \"Oh, hello!\",\n",
    "        \"What's your name?\",\n",
    "        \"My friends call me World\",\n",
    "        \"Hello World!\"\n",
    "    ]\n",
    ")\n",
    "len(embeddings), len(embeddings[0])"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-08-26T09:32:14.879574700Z",
     "start_time": "2023-08-26T09:32:10.744392500Z"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "outputs": [
    {
     "data": {
      "text/plain": "[0.005387211957276042,\n -0.0005941777859814659,\n 0.03892524773846194,\n -0.00297914132073842,\n -0.008912666382268376]"
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "embedded_query = embeddings_model.embed_query(\"What was the name mentioned in the conversation?\")\n",
    "embedded_query[:5]"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-08-26T09:32:17.733179200Z",
     "start_time": "2023-08-26T09:32:17.372139900Z"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "outputs": [],
   "source": [
    "# 嵌入可以存储或临时缓存，以避免需要重新计算它们。\n",
    "from langchain.storage import InMemoryStore, LocalFileStore\n",
    "from langchain.embeddings import OpenAIEmbeddings, CacheBackedEmbeddings\n",
    "from langchain.document_loaders import TextLoader\n",
    "from langchain.embeddings.openai import OpenAIEmbeddings\n",
    "from langchain.text_splitter import CharacterTextSplitter\n",
    "from langchain.vectorstores import FAISS\n",
    "underlying_embeddings = OpenAIEmbeddings()\n",
    "fs = LocalFileStore(\"./cache/\")\n",
    "\n",
    "cached_embedder = CacheBackedEmbeddings.from_bytes_store(\n",
    "    underlying_embeddings, fs, namespace=underlying_embeddings.model\n",
    ")"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-08-26T09:35:01.521962900Z",
     "start_time": "2023-08-26T09:34:59.197867200Z"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "outputs": [
    {
     "data": {
      "text/plain": "[]"
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#缓存在嵌入之前为空\n",
    "list(fs.yield_keys())"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-08-26T09:35:18.744268200Z",
     "start_time": "2023-08-26T09:35:18.718319600Z"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "outputs": [],
   "source": [
    "#加载文档，将其拆分为块，嵌入每个块并将其加载到矢量存储中。\n",
    "raw_documents = TextLoader(\"documentstore/state_of_the_union.txt\").load()\n",
    "text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)\n",
    "documents = text_splitter.split_documents(raw_documents)\n",
    "#create the vectorstore 创建矢量存储\n",
    "db = FAISS.from_documents(documents, cached_embedder)\n"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-08-26T09:36:21.954703Z",
     "start_time": "2023-08-26T09:36:21.594258Z"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "outputs": [],
   "source": [
    "#如果我们尝试再次创建 vectostore，它会快得多，因为它不需要重新计算任何嵌入。\n",
    "db2 = FAISS.from_documents(documents, cached_embedder)"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-08-26T09:36:35.214886900Z",
     "start_time": "2023-08-26T09:36:35.196842300Z"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "outputs": [
    {
     "data": {
      "text/plain": "['text-embedding-ada-002948b99a3-567e-5f0f-9ccb-d68ed5b060d9']"
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#缓存中现在有一些内容\n",
    "list(fs.yield_keys())[:5]"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-08-26T09:36:44.770045600Z",
     "start_time": "2023-08-26T09:36:44.760037400Z"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "outputs": [],
   "source": [
    "#在内存中\n",
    "store = InMemoryStore()\n",
    "underlying_embeddings = OpenAIEmbeddings()\n",
    "embedder = CacheBackedEmbeddings.from_bytes_store(\n",
    "    underlying_embeddings, store, namespace=underlying_embeddings.model\n",
    ")\n",
    "embeddings = embedder.embed_documents([\"hello\", \"goodbye\"])"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-08-26T09:37:19.164780900Z",
     "start_time": "2023-08-26T09:37:18.561607500Z"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "outputs": [],
   "source": [
    "#我们第二次尝试嵌入嵌入的时间只有 2 毫秒，因为嵌入是在缓存中查找的。\n",
    "embeddings_from_cache = embedder.embed_documents([\"hello\", \"goodbye\"])"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-08-26T09:37:33.168470Z",
     "start_time": "2023-08-26T09:37:33.147473700Z"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "outputs": [
    {
     "data": {
      "text/plain": "True"
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "embeddings == embeddings_from_cache"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-08-26T09:37:40.086042700Z",
     "start_time": "2023-08-26T09:37:40.063713500Z"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [],
   "metadata": {
    "collapsed": false
   }
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
